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ABSTRACT: The disconnect between data collection and analysis across academic and 
administrative units within institutions of higher education makes it challenging to incorporate 
diverse data into curricular design. Understanding the factors related to student retention and 
success is unlikely to occur by focusing on only one unit at a time. By promoting course design 
informed by data from diverse units, we are more likely to understand important connections 
that could encourage more effective and holistic change. Facilitating evidence-based course 
design should begin with a faculty-driven process to identify how best to explore data across 
units for integration into the traditional course/curricular design. Our paper describes how data 
from institutional, learning, and what we call "developmental" analytics can be incorporated into 
course and curricular design by using a purposefully built analysis tool that permits the 
exploration of data relevant to course/curriculum design. This Browser of Student and Course 
Objects (BoSCO) is being built in a faculty-driven process and can be used as a bridge between 
the analytics space and the course/curriculum design environments, therefore encouraging 
faculty to use analytics for course and curricular design. 
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1. INTRODUCTION 


Day after day, day after day. 

We stuck, nor breath nor motion; 

As idle as a painted ship 
Upon a painted ocean. 

Water, water, every where, 

And all the boards did shrink; 

Water, water, every where. 

Nor any drop to drink. 

—Samuel Taylor Coleridge, 

"The Rime of the Ancient Mariner" 

There has been a clear call-to-action to improve the state of education at the undergraduate level 
(Boyer Commission, 1998; National Research Council, 1996; National Science Foundation, 1996; 
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Executive Office, 2009). Many departments face significant pressure from accreditors, administrators, 
and, for public institutions, state legislators, to detail student learning within departments, majors, or 
programs (U.S. Department of Education, 2000). Substantial resources and effort have led to the 
development and implementation of a striking array of evidence-based practices within classrooms, 
curricula, and campuses. However, the majority of these initiatives have been developed and 
implemented independently with little intentional communication among disciplines or across academic 
and administrative units (Labov, Singer, George, Schweingruber, & Hilton, 2009; Henderson, Beach, & 
Finkelstein, 2011). Given that these independent efforts have led to moderate success (Boyer 
Commission, 1998; Handelsman, Miller, & Pfund, 2007), it seems reasonable to hypothesize that a larger 
effect on education might be achieved with a more synergistic approach based on a concerted and 
collaborative effort on the part of educators, researchers, staff, and administration. Indeed, the most 
effective approach to changing undergraduate instruction involves designing long-term strategies that 
are compatible with the complex and dynamic nature of colleges and universities (Henderson et al., 
2011). At the same time, however, it is challenging to find a practical, systematic way to connect these 
individual, separate practices into unified processes. Such processes have the potential to promote 
student retention and success by permitting the exploration of data immediately relevant to designers 
of individual courses and also to groups or individuals responsible for higher-level programmatic 
curricular design and evaluation. 

Solutions to expansive problems like student retention and success are unlikely to be reached by 
independently adjusting factors in separate units of an institution. To begin to facilitate reaching the 
goals of increased student retention and success on a campus or a program, we must promote it in the 
classroom through course design informed by diverse types of data typically collected disparately across 
an institution. Like water in "The Rime of the Ancient Mariner," data seems to be everywhere but in 
practice, effectively using that data to inform curricular design presents an enormous challenge. As 
others (Henderson et al., 2011; Buerck & Mudigonda, 2014) have shown, top-down approaches to 
academic analytics or curricular change initiatives are rarely successful. In this paper, we describe a 
bottom-up approach that we have implemented in an effort to facilitate faculty usage of data in course 
design. This approach was based on the belief that building programs and courses informed by data 
requires that the people designing the programs and courses, generally faculty, are part of any effort to 
gather data, define questions, and design the necessary tools. We also believe that this approach will 
have a ripple effect radiating from the course level through the program level by providing data not only 
useful to course designers but also to program designers. 

Our approach, which drew heavily from our current understanding of academic analytics, design theory, 
and the principles of institutional change, began with talking to all faculty within our interdisciplinary 
department. Through these discussions, it became very clear that faculty wanted to collect and analyze 
data not typically associated with course learning objectives, but variables typically of interest to those 
in student development, like student attitudes, curiosity, or self-awareness. As we explain below, the 
high faculty demand for this type of data suggested to us that the endeavour of academic analytics 
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might benefit from the inclusion of data from institutional, learning, and what we call “developmental" 
analytics. Our process, which has to date included one semi-structured focus group, two formal 
meetings, two surveys, and multiple informal feedback sessions, also revealed that faculty would be 
more likely to use analytics tools that permit them to explore the datasets they think are relevant. As a 
result, a principal outcome of our bottom-up process has been the development of a tool that gives 
faculty the ability to share and browse datasets related to the areas of institutional, learning, and 
developmental analytics. It is our hope that use of this tool not only facilitates data-driven course design 
but also provides a refined lens to inform future research questions related to understanding the 
relationship between course design and student success and retention. 

2. COURSE AND CURRICULAR DESIGN USING DATA 

Work by others has clearly demonstrated the value of adopting a synergistic view of incorporating 
diverse types of information when exploring variables related to student retention and success. For 
example, the research of Tinto (1997) and others (Antonio, 2004; Buckingham Shum & Ferguson, 2012; 
Severiens & Schmidt, 2009) relies heavily on diverse datasets and demonstrates that student 
involvement and participation in schools are important factors in student retention. In fact, in one study, 
Tinto (1997) concludes that encouraging greater involvement in academic and social activities positively 
affects student learning and retention. This study strongly suggests that the practices and policies across 
an institution are critically interrelated when it comes to the relationship between student retention, 
academic experiences, and social development and involvement (Tinto, 1997). 

Though it presents theoretical and practical challenges, we must move beyond simply knowing that data 
from a variety of units across an academic institution are important for understanding correlations with 
student success and retention, to an environment in which the outcome of analyzing these different 
datasets can be effectively incorporated into course and curriculum design. At a foundational level, this 
transition requires an understanding of what "analytics" means across the academic and administrative 
units of an institution, a design framework in which to use data effectively for course design, and tools 
that permit timely and informative analysis of data relevant to course design. Below we clarify our 
understanding of each of these requirements as they apply to facilitating evidence-based curricular 
design. 

2.1. Academic Analytics 

Understanding exactly what the term "analytics" means in academia can be confusing because the 
terminology is connected to but not always aligned with that used to describe the institutional and 
course goals being explored with analytics (Siemens & Long, 2011; Oblinger, 2012a). However, at a 
general level, we can define analytics as "the use of data, statistical and quantitative methods, and 
explanatory and predictive models to allow organizations and individuals to gain insights into and act on 
complex issues" (Oblinger, 2012b, para. 2). "Academic analytics" refers to cases in which analytics are 
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applied within academic settings and includes two subtypes of applied analytics called "institutional 
analytics" and "learning analytics." At the institutional/administrative level, analytics is often used to 
understand factors that relate specifically to helping "run the business of the higher education 
institution" by attempting to predict student success and retention rates (Oblinger, 2012b, para. 2). The 
related, but to date conceptually separate term "learning analytics" is best understood as analytics that 
focus "on students and their learning behaviours, gathering data from course management and student 
information systems in order to improve student success" (Oblinger, 2012b, para. 2). Both institutional 
and learning analytics align well with the goals of academic analytics within an institution, as they 
facilitate understanding of student success and retention from slightly different perspectives (Figure 1). 


Subtype of Academic Analytics Examples of Data Used Level Goals 



Figure 1. Summary of academic analytics, institutional analytics, and learning analytics. 

2.2. Course Design Model 

Facilitating a strong connection among course design, data collection, and data analysis represents a 
tremendous opportunity to increase both the efficiency and effectiveness of course and curriculum 
design in undergraduate education. Information gathered by faculty members using well-designed 
learning analytics tools would facilitate the use of the available data as part of a modified Analysis, 
Design, Develop, Implement, and Evaluate (ADDIE) instructional design process (Molenda, 2003). Using 
this process, an instructor would design, develop, and implement a course, and then analyze data in a 
rigorous way to assess what worked, what did not, what should be maintained, and what should be 
modified or discontinued (Figure 2A). However, traditional course design generally deviates from this 
process, and is best described as a modified ADDIE process in which the design and delivery of courses is 
mostly (or only) informed by faculty intuition or unstructured observations about how previous courses 
went. In fact, exploring the "data" in this case is often limited to an instructor's quick perusal of overall 
grade distribution, reaction to anecdotal evidence, like verbal feedback from students after a class, or 
visual perception of students' interest during the class. As shown in Figure 2B, the major pathway (large 
arrow) from "implement" to "design" bypasses "analyze data." Large data sets, in combination with 
tools and methods that allow for easy analysis of the data, have the potential to allow faculty to add 
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depth to the evaluation phase of typical instructional design through the analysis of data relevant to 
curriculum design, which then informs design and delivery (Figure 2C). However, as indicated by the 
presence of the dotted arrow, even the most robust data sets do not completely erase the need for the 
"art" of teaching. Anecdotal evidence, including informal student feedback or instructor observations of 
student engagement during class, does and should provide useful clues to instructors in terms of 
modifying instruction, but should be secondary to more structured and rigorous assessment methods. 



ADDIE 
In Theory 




ADDIE 

Traditionally Applied 


ADDIE 

Proposed 


Figure 2. Schematic of the Analyze, Design, Develop, Implement, and Evaluate (ADDIE) model in 

theory and as applied in curriculum design. 

A) Theoretical ADDIE model. B) Traditional curriculum design strategy with the tendency to bypass 
rigorous data analysis in the design process as denoted by the large arrow between "Implement" and 
"Design." C) A modified approach that includes a more intentional use of data in the design process as 
denoted by larger arrows between "Implement," "Analyze Data," and "Design." 


2.3. Existing Tools 

In order to put data back into the ADDIE process, many groups are working to design and implement 
learning analytics tools for e-learning (c.f., Ali, Hatala, Gasevic, & Jovanovic, 2012; Buerck & Mudigonda, 
2014; Ferguson, 2012; Macfadyen & Dawson, 2010). These tools and strategies have made some gains in 
helping faculty visualize student access and activities in Learning Course Management Systems, 
especially for online courses (Ali et al., 2012; Ferguson, 2012). However, the results from these programs 
also serve to highlight two major problems faced by the emerging field of learning analytics. Morris, 
Finnegan, and Wu (2005) and Macfadyen and Dawson (2010) both found that their variables, which 
similarly include visits to an online course website and time on task, explained only about 30 percent of 
variance in students' final grades. As Morris et al. admit, their study confirms the obvious: that "students 
who are more engaged with the content and discussions in an online course will persist and complete 
successfully" (2005, p. 229). This analysis might be perceived as a first step in learning analytics, and is 
therefore necessary and meaningful, but complete analysis requires researchers to move beyond this 
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"low hanging fruit" and seek variables and questions that faculty identify as meaningful, and provide 
guidance for them with respect to course design. 

To date, research in this area has started to explore very broad questions from a wide variety of 
perspectives, but with limited tools and data. For example, course management systems might track 
whether students download a particular reading, if they ask questions in online forums, or how much 
time they spend on certain online tasks (c.f., Ali et al., 2012; Macfadyen & Dawson, 2010). In these 
cases, researchers are limited to the data available in the course management system they use or by 
restrictions imposed by top-down approaches to data collection (Buerck & Mudigonda, 2014). Questions 
are therefore often driven by what data is easy to collect rather than what data is pedagogically 
meaningful. In fact, feedback from faculty about LOCO-Analyst revealed that most respondents did not 
find the information useful (Ali et al., 2012; Ferguson, 2012). A related problem is synthesizing the data 
in a way that will encourage faculty to use them (Ali et al., 2012) or creating a process that achieves buy- 
in from relevant stakeholders (Buerck & Mudigonda, 2014). 

2.4. Institutional Change 

Even with these tools and processes, implementing evidence-based educational practices requires 
careful planning and institutional self-reflection (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). 
Individual departments or programs seeking to implement such change in order to demonstrate the 
worth of their curriculum may face considerable resistance from stakeholders, like faculty members. In 
order to overcome these challenges, we have chosen to follow empirically based principles of change, 
such as those outlined by Adrienne Kezar and others (Henderson et al., 2011; Kezar & Eckel, 2002; Kezar, 
2001; Lindquist, 1978). Kezar (2001) provides a detailed list of research-based principles of change. 
Some of these principles are beneficial attitudinal factors, like being open and aware of politics at the 
institution, and realizing that change is a disorderly process. Others detail concrete strategies that guide 
the process and goals, including engaging in organizational self-discovery, facilitating "interaction to 
develop new mental models and sense-making" (Kezar, 2001, p. 118), and creating a roadmap for 
strategic change (Kezar, 2001; Lindquist, 1978). These activities mirror change strategies laid out 
elsewhere, like engaging in a collaborative process, motivating a vision and mission, and developing 
support structures (Henderson et al., 2011; Kezar & Eckel, 2002). Because institutional changes relating 
to the curriculum directly affect faculty members, they are an important group to incorporate into the 
change process. In fact, the American Sociological Association (2005) emphasizes the role of faculty in 
instituting changes suggested by the empirical assessment of the goals, and a recent study also found 
that 78 percent of university presidents believe that faculty should drive institutional change and 
innovation (Selingo, 2013). 
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3. PLACE AND PROCESS 

As is increasingly the case across academia, the Center for Learning Innovation (CLI) within the 
University of Minnesota Rochester (UMR) is committed to bringing together faculty, IT (Information 
Technology), and administration to share data, knowledge, and resources in an effort to provide the 
best possible learning environment for students (Neuhauser, 2012; Neuhauser & Weber, 2011). The 
characteristics and qualities of the environment in the CLI (our department) make it an ideal place to 
develop, test, and implement tools and processes prior to distribution of those tools to academic 
environments potentially too complex for the early stages of such endeavours. First, faculty from across 
a variety of disciplines are housed in one department, and all tenured/tenure-track faculty are required 
to do research on learning as part of their tenure and promotion requirements. As a result, faculty in our 
department, which contains members trained in literature, public health, sociology, biology, chemistry, 
mathematics, physics, and philosophy, are accustomed to discussing a variety of different disciplinary 
and cross-disciplinary indicators for assessment and research. In fact, we have already approved 
Institutional Review Board protocols (#1008E87333; #0908S71602) in place for human subjects research 
that includes collecting student grades and course work across all courses offered in our department. 
Second, much student data is already stored in relatively central locations. All instructors use our in- 
house curriculum management system, iSEAL (intelligent System for Education, Assessment, and 
Learning ), which contains student assignments and course grade books (University of Minnesota 
Rochester, n.d.). We are also fortunate that, as part of the University of Minnesota system, we have the 
ability to share and access assignments and activities stored within both Google Drive and Moodle. 
Finally, the small size of our faculty and general openness to sharing de-identified data means it would 
be relatively easy to solicit input from the faculty, and to test preliminary data analysis tools across a 
wide array of disciplines. Because of our highly integrated core curriculum, we can analyze how 
performance on specific assignments in a course correlates with another assignment in a different 
course. 

Despite these strengths, our faculty are still challenged by the practical hurdles of compiling and 
exploring data from a variety of courses, surveys, and databases. We believe, in line with the tenants of 
institutional change, that one of the best ways to encourage faculty to connect analytics with course and 
curriculum design at our and other academic institutions is to create a process that allows faculty to 
identify variables that are significant and useful to them. Our ultimate goal is to facilitate faculty 
implementation of evidence-based course design. In line with principles of institutional change, a 
priority was to keep faculty at the centre of and driving the process. 

After a variety of informal discussions with faculty in our department, the authors applied for and 
received an internal grant from the University of Minnesota Provost's Office (from the “Enhancement of 
Academic Programs Using Digital Technology" program) to develop a data-browsing tool. Following the 
receipt of that grant, our first step was to administer a survey and convene a focus group, run by the 
authors and attended by nine other tenure-track faculty in the department, to explore the types of 
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questions each faculty member was interested in exploring, and what kinds of data they would need to 
explore those questions. This survey and focus group allowed us to engage in self-discovery, as well as 
facilitate interaction and help all faculty members make sense of the goals of developing this tool (Kezar, 
2001). In this focus group, faculty members outlined a list of indicators associated with pedagogical 
questions relevant to the evaluation and design of their courses. The faculty members also discussed the 
location (iSEAL, Moodle, Google Drive, admissions, student affairs, etc.) and accessibility of the data 
associated with each indicator. The authors then summarized and generated a list of questions with the 
associated indicators, and identified the potential mechanisms that could be used to allow them to 
access data through BoSCO. The authors then met to finalize the prioritized list as well as begin to 
discuss possible statistical tools that might be useful. The final prioritized questions, associated 
indicators, relevant databases, and suggested statistical tools were then used to build a mock-up of the 
learning analytics tool (BoSCO). In other words, instead of starting with the data and identifying 
questions to ask from there, we started with the research questions, and then sought to identify how to 
collate and browse the data in a way that allowed faculty to draw useful connections. 

4. OUTCOMES 

4.1. Developmental Analytics 

The first outcome of our faculty-driven process was that we learned faculty believed data about student 
attitudes and affective traits were critical for their research on curriculum and curricular design. This 
type of data is exemplified by student developmental outcomes, like those adopted by the University of 
Minnesota (2011). Understanding the way students emotionally grow and develop during college is 
valuable to understanding curricular changes that could or should be made (Baxter Magolda, 2003; 
Oblinger, 2012a). As such, we suggest the addition of another category, developmental analytics, as a 
third subtype of analytics in the academic space (Figure 3). This concept is related to Buckingham Shum 
and Ferguson's (2012) Social Learning Analytics (SLA), but where SLA focuses on learning in a 
"participatory online culture," developmental analytics is broader, encompassing student experiences, 
skills, and affective and personality traits, both in and outside of the classroom. Using this type of 
analytics, researchers would gather and analyze data related to the development of students' 
dispositional traits and skills like curiosity, resilience, independence, interdependence, and self- 
awareness. Developmental analytics conceptually aligns administrative goals, learning outcomes, and 
developmental outcomes so that diverse types of data and analyses can be used by those engaging in 
course and curricular design. 

The different types of analytics used in academic institutions (institutional, developmental, and learning) 
deal with some distinct types of information, but some of the information that falls within each category 
is shared. In fact, we argue that the information and analyses shared by each of the three subtypes 
would be extremely valuable to course and curriculum designers in their efforts to construct learning 
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environments that serve to increase student retention, teach specific content, and promote the 
development of soft-skills (Oblinger, 2012a). 


Subtype of Academic Analytics Examples of Data Used Level Goals 



Figure 3. Models of subtypes of academic analytics with the inclusion of developmental analytics as a 
form of analytics, valuable conceptually and practically, for curriculum design. 


4.2. Browser of Student and Course Objects (BoSCO) 

Our interaction with faculty led us to believe that the best way to move forward once these variables 
were identified was to create a tool that allowed faculty a meaningful way to interact with student- and 
course-related data in a dynamic fashion. Based on faculty response to date, it is our hope that this 
interactive process — in which we facilitate faculty buy-in and include easy to use analysis tools in 
BoSCO — allows for a bridge to be built between the analytics space and the course/curriculum design 
environments (Figure 4). Specifically, our hope is that a tool that uses data defined as relevant by course 
and curricular designers will allow us and others to evolve from a model where data relevant to 
institutional and learning analytics exists but is largely unused by designers who default to a design 
model that replaces data with only course grades, intuition, and/or anecdote (Figure 4A). By 
incorporating data from developmental analytics and providing a tool to browse data from all three 
subtypes of academic analytics, it is hoped that our curricular designers and others can move to a design 
process that brings data analysis into the design process but does not eliminate the possibility of 
including information gathered by experience and/or intuition (Figure 4B). We also wanted to create a 
tool that was open and available, so that it could be useful to faculty members and departments at 
other campuses and institutions. 
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Curriculum 

Subtype of Academic Analytics Examples of Data Used Design 




B) 

Subtype of Academic Analytics 


Examples of Data Used 



Curriculum 

Design 


BoSCO 



Figure 4. Schematic of the relationship between analytics and curriculum design. 

A) Under the traditional curriculum design strategy and with current conceptual distinctions in analytics. 
Note the lack of connection between analytics and curriculum design, and the tendency to bypass 
rigorous data analysis in the design process as denoted by the large arrow between “Deliver" and 
"Design." B) A modified approach that incorporates developmental analytics and includes a more 
intentional use of quantitative data in the design process as denoted by larger arrows between 
"Implement," "Analyze Data," and "Design." Note that BoSCO is an example of a tool that bridges the 
analytics and design areas by facilitating the inclusion of relevant data and analyses to curriculum 

designers. 
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The information gathered from the faculty focus groups was first used to generate a conceptual model 
of the foundational components of the BoSCO tool (Figure 5). This model included the primary data 
sources that contained the types of information that the faculty found the most relevant, including the 
ability to explore survey data about student attitudes or other student-development-related 
information (Figure 5, lower left box). Also included in the model were the basic display, user interface, 
and analysis elements necessary to begin the build out of the project. 



(query) ,(raw data) 


Browser of student data 


Graphical 
and data 

download options 


Select assignments: 
Course 1 
Assignment 1 
Assignment 2 

Course 2 

(...) 


Graphical 

Display 


Select topics/skills: 


Select 

Topic 1 


institutional data: 

Topic 2 


Gender 



^Credits 

Skill 1 


GPA 

Skill 2 


ACT 


Statistical 

analysis 

methods 


Data display 
and 

manipulation 

area 


Data select 
area 


Figure 5. Conceptual model of BoSCO showing the basics of the graphical user interface as well as the 
specific data sources determined to be relevant based on faculty input. 

The diverse types of data requested by faculty require that data input into BoSCO be uniform but 
flexible. To promote a uniform data structure, BoSCO requires comma separated value (.csv) files with a 
specific file format that can be generated automatically by downloading the files directly from iSEAL or 
manually by downloading a template .csv file and copying and pasting data into the template prior to 
uploading to BoSCO. To accommodate the need for flexibility, the current version of the tool includes an 
introductory window for uploading data in the form of "course," "student performance," and/or 
"demographic" .csv files. Permitting files to be uploaded manually allows the tool to be more flexible in 
that it means researchers can include data from outside of iSEAL. In the future, we plan to explore the 
possibility of downloading BoSCO formatted files directly from other learning management systems 
(e.g., Moodle) to reduce the need for manual formatting. Multiple course-performance data files can be 
uploaded into BoSCO, which can then be used to compare individual students' performance across 
multiple courses. 
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Permitting multiple types of data to be uploaded and explored by BoSCO also means that each faculty 
member uploading data is responsible for having the appropriate institutional access and, when 
required, Institutional Review Board (IRB) approval. Currently, student privacy and consent can be 
managed in two ways. In the first, the Course Management System that stores the student data also 
keeps the record of students who gave their consent under the appropriate IRB protocol. This is the 
strategy adopted in our department where iSEAL merges and filters the necessary data and de-identifies 
the students before faculty download the .csv files in BoSCO format. By doing so, faculty can combine 
several datasets belonging to different researchers without dealing with any identifiable data. A second 
way to manage privacy and consent is by individual faculty members manually de-identifying students in 
the dataset(s) associated with each of their courses and cross-referencing the students with a list of 
students who consented under the appropriate IRB protocol. Based on feedback from our faculty group, 
this second process is moderately useful for the basic browsing of individual courses but becomes very 
labour intensive when trying to align student lists across courses or with data collected outside of 
courses. In this case, the first approach is preferred. However, if data is not being used for research but 
just for internal quality control or curriculum design, in the U.S., no institutional review or consent is 
required. 

The BoSCO web platform was designed as a lightweight client-side application that can run on any 
device. The web platform is a set of JavaScript libraries that read the files uploaded by the user and 
create a session-key. It does not call any third-party servers. Once the session is over, the user decides 
whether or not to save the session-key and the data associated to it. Because of its lightweight 
characteristics, rather than focusing on computationally demanding statistical analysis, BoSCO focuses 
on the browsing, filtering, and graphical display of data using JavaScript graphical libraries such as Flot 
(Laursen & IOLA, 2013) and d3 (Bostock, 2013). 

4.3. How BoSCO can be used to Explore Data 

In order to bridge institutional, academic, and developmental analytics, our process indicates that 
faculty want a tool that goes beyond exploring de-identified grade books. In addition to exploring 
variables and relationships from the curricular level, through the course level, and down to the level of 
assignments, faculty wanted to be able to evaluate student performance in courses against other types 
of data, such as demographic information and attitudinal or survey data. In the short term, in order to 
make practical this type of data exploration, faculty can upload course grade books, survey, and other 
types of data into the current "beta" version of BoSCO. Once uploaded, the user can begin to explore 
the data from an assignment, course, or semester. In the longer term, we plan to automate this process 
so that users can employ BoSCO to upload data directly from common course management systems, like 
Moodle. 
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As shown in Figure 6, users can begin their exploration by evaluating relationships among a variety of 
variables with some basic statistics. The user can select a course-level analysis (Figure 6A), which 
permits a high-level view of how students are performing in one course relative to another (R 2 ) and can 
include all courses in a given program. For the purpose of demonstration, we show the results of a 
comparison of final grades among two different biology courses (BIOL 2331, 2332: Anatomy and 
Physiology I, II), two different chemistry courses (CFIEM 2331, 2333: General Chemistry I, II), and an 
introductory sociology course (SOC1571: Sociology) completed in 2011 and 2012 in our program. The 
semester-level analysis (Figure 6B) shows correlations in student performance in the same five courses 
over multiple semesters. The assignment-level shown in Figure 6C shows correlations in student 
performance on a variety of selected assignments completed in CFHEM2331 and SOC 1571. 


Graphical User 
Interface 


Level of 
Analysis 


■> Output 



Figure 6: Sample analysis by level. This particular output shows A) a matrix of R 2 values for student 
course grades across 5 courses (BIOL 2331, 2332; CHEM 2331, 2333; SOC 1571), B) R z values for 
student course grades for the same 5 courses and across multiple semesters, and C) R 2 values for 
student grades on all assignments in CHEM 2331 and SOC 1571 in a single semester. 

Should the user want to probe deeper than the big picture view offered by the analysis shown in Figure 
6A, they can select (click) a specific cell in the correlation matrix, which then takes them to a different 
interface that permits exploration of variables in two specific courses. As shown in Figure 7, for example, 
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clicking on the cell containing the R 2 value calculated for SOC 1571 and CHEM 2331 allows the user to 
choose to view a bar chart or scatterplot of student performance with all students (no filter), or filtered 
by a variety of variables. In this case, the analysis filtered by sex shows only the final course grades for 
females enrolled in both courses, the analysis filtered by race shows only the final grades of students of 
colour. From here, the user can probe even deeper, by choosing specific assignments to compare using 
the filtered data. In this case, the assignment-level analysis shows the bar charts and scatterplots for a 
group paper assignment in sociology, and a homework quiz from chemistry. 


BIOL2.W BI0L:M: Chra23M Chra:.W SOC1571 



Filter 


None 

* 


Sex 

* 


Race 

* 


| SOC1571 Final Grade 
j CHEM2331 Final Grade 


| SOC 1571/Female 
| CHEM2331/Female 


JL . _ Jhl 


I SOC1571/Students of Color 
| CHEM2331/Students of Color 




I 

Semester/ 

Assignment 

I 


I 

Semester/ 

Assignment 

I 


■ SOC1571/female/Paper 

■ CHEM2331/female/ 
Homework 



HsOC1571/Studentsof Color/Paper J 
HCHEM2331/ Students of 
Color/Homework 



• vnjp* 

X SOC1571/Female/Paper 
Y CHEM2331/Female/ 

Homework 



X SOC1571/Students of Color/PaDer 
Y CHEM2331/Studentsof 
Color/Homework 


Figure 7. The user can further explore matrix results using bar charts or scatterplots, and filtering by 
demographic variables like sex or race, or filtering by specific assignments. The straight diagonal line 
in these scatterplots represents x=y, and can be used to compare whether students performed better 

in the y-axis category or the x-axis category. 
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This tool has the potential to be useful at the course, department, or program level to explore student 
success and retention. A faculty member or department head could identify courses or assignments that 
are challenging for all students, or for a subset of students. This tool could help answer a variety of 
questions: are students who take specific courses early in their college career more successful in the 
program? Are women, students of colour, or first-generation college students particularly likely to 
struggle with a specific course or set of assignments? Answers to these questions might guide advising, 
sequencing of required courses, or revision of assignments. 
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Figure 8. BoSCO outputs comparing survey and performance data. In this case, a cursory analysis 
indicates little correlation between students' social self-efficacy and final course grades in a variety of 

courses. 


With the goal of making this tool useful for faculty, it was important that BoSCO have the additional 
capability of allowing the exploration of attitudinal or dispositional data along with course performance 
data (Figure 8). Exploring these types of data together can be extremely useful. In our degree program, 
for example, students engage in a significant amount of group work, the results of which may make up a 
large portion of their final course grade. While there is evidence to support this pedagogical approach 
(Johnson, Johnson, & Stanne, 2000; Slavin, 1991), learning more about the relationship between course 
performance and how students rate their social self-efficacy could be very informative for supporting 
the model in our environment. Faculty here wanted to know whether requiring a significant amount of 
collaborative work is correlated with lower grades for students who were less outgoing, shyer, or with 
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lower self-efficacy in social settings; if so, then alterations in the pedagogical approach might be 
necessary. 

To measure this personality characteristic, we needed data that fell outside the area of traditional 
course performance and closer to the area of student development. To that end, we looked to the work 
of Sherer et al. (1982), which supported the possibility of splitting the results of a self-efficacy 
assessment into two dimensions. One dimension is an indication of a global sense of self-confidence in a 
person's abilities to achieve future success, which Sherer et al. (1982) coined "general self-efficacy." The 
second dimension includes individuals' belief in their competence in social situations, or social self- 
efficacy (Sherer et al., 1982). Social self-efficacy has been found to be correlated with a perceived social 
regard, the likelihood of instigating interactions with others, and overall competence in social situations 
(Connolly, 1989). Therefore, after giving most of the students in our program a short social self-efficacy 
survey (consistent with approved IRB protocols), the data were uploaded to BoSCO. Using BoSCO, the 
user easily obtained R 2 and scatterplots to explore the correlation between social self-efficacy and final 
course grades in Anatomy and Physiology, General Chemistry I and II, and Sociology. In this case, R 2 and 
a visual analysis of scatterplots indicate a low correlation between social self-efficacy and final course 
grades, which supports the use of collaborative learning strategies since they appear not to be harmful 
for those students with lower social self-efficacy. From this analysis, a combined data file that includes 
the data from a variety of sources can be downloaded for more in-depth and rigorous analysis in a 
statistical analysis package, but BoSCO allows for initial analyses that can inform these research 
questions. 

5. LIMITATIONS 

Though we have made significant progress in creating a process to encourage and facilitate integrating 
empirical evidence into course design and revision, there are limitations. Our process appears to be 
working well within our environment but we have a relatively small number of faculty within a unique 
educational environment. For example, as mentioned above, our tenure code requires tenured/tenure 
track faculty to engage in research on teaching and learning. What faculty wish to research here, their 
willingness to share data, and the time they are willing to expend doing that research, may be quite 
different from other departments. Therefore, identifying what data faculty want may be simpler here 
than in larger departments. Flowever, there are many departments that face the problem of defending 
their worth and/or curriculum to administration or accreditors, and such a tool may be useful to them 
(U.S. Department of Education, 2000). Even with participation from all faculty, much work has to be 
done to develop communication pathways among different units of an institution in order to identify 
and access data, and to ask questions useful across the entire institution. This can be quite time 
consuming and is potentially fraught with a variety of political hurdles (Flenderson et al., 2011; Buerck & 
Mudigonda, 2014). 
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The tool that has emerged from our process also has significant limitations in its current form. Manually 
uploading spreadsheets is more labour intensive than is desirable in the long-term, but it does eliminate 
two problems. One, BoSCO does not have to deal with political problems regarding who owns the data 
being browsed. As the one compiling the spreadsheets, the researcher and their collaborator(s) are 
responsible for complying with IRB and FERPA regulations. Two, because it does not need to connect to 
any particular server to download data, BoSCO works with any Course Management System. As long as 
the data being uploaded are in the right format, BoSCO can browse them. Furthermore, BoSCO is 
intended to be a way to explore data and not a robust quantitative data analysis tool. To gain more 
analysis strength we are designing BoSCO to give users the ability to download data in formats 
commonly used by quantitative analysis software, like R or SPSS. 

6. CONCLUSION 

As those in academia push forward in pursuit of discovering the most effective ways to inform best 
practices, it is imperative that processes and tools are developed that permit curriculum designers to 
acquire and explore data relevant to student learning, development, and retention. Such tools would 
also be extremely useful to departments seeking to present evidence of a quality program to 
administrators, accreditors, and others. To innovate, institutions must expand beyond relatively narrow 
analyses and incorporate diverse datasets from across the institution to provide a more holistic vision of 
students. In other words, students exist outside the classroom, and understanding the "whole student" 
— inside and outside the classroom — provides us with a more complex and complete picture of 
student performance. While many institutions have begun to require teachers in higher education to 
provide data regarding the learning that takes place in their classrooms, these datasets provide an 
incomplete picture of student success. Institutions of higher education need to build structures that 
facilitate communication across the institution to permit curriculum designers access to data that would 
complement the data they already have on student performance in the classroom. With a more 
complete picture, faculty and administrators have a stronger foundation from which to innovate in 
terms of both educational strategies, and ways to explore and document a variety of factors that affect 
student learning, development, and retention. 

In most places of higher education, integrating evidence-based practices involves a change from the 
norm. The literature on institutional change offers some effective and empirically based strategies to 
alter educational practices effectively. At our institution, our process has led to us to identify two needs, 
both articulated by faculty. First is the need to add developmental analytics to institutional and learning 
analytics as subtypes of academic analytics, and second, faculty need to have a tool to explore the data 
associated with all three subtypes of academic analytics. If designed and developed properly, with 
appropriate input from stakeholders, it is our belief that these or similar tools and processes can 
facilitate asking and answering questions that designers, instructors, and administrators have about 
student achievement across an entire curriculum. 
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It is not our intention for BoSCO to be an end in itself. Just as narrow analyses focusing on only one slice 
of a student's experience in an institution is unlikely to yield data that can guide institutional change, 
one tool cannot be the solution to all institutional problems. Because BoSCO was developed by faculty 
to help them explore their research questions, its strength lies in its flexibility to incorporate data from a 
variety of sources and provide a way to begin exploring the data. BoSCO would be a first step, a place 
where faculty and others could begin to explore data in an intuitive way, begin to ask questions, and 
begin to understand how a variety of disparate data sets do or do not fit together into a whole. Because 
it was designed to start with questions and not specific data, the driving force will always be the creative 
vision of the user. BoSCO is a lens that helps the user understand both the big picture, and the way that 
subsets of students fit within that bigger picture. 

Forming a functional connection between analytics and curriculum design is a complicated and daunting 
endeavour made more challenging by the continuous growth of data available to curriculum designers. 
Awareness of the existence of the data is not enough. To draw useful conclusions about what data is or 
is not correlated with best practices, curricular designers need tools that allow them to navigate and 
explore the ever-growing space of usable data in the way a telescope is used to explore celestial objects. 
However, unlike space, data is not continuous, nor are all datasets equally relevant to the curriculum 
designer or education researcher. These tools should allow interested designers and researchers the 
opportunity to navigate relevant datasets intentionally, categorize elements within those datasets, and 
inform hypotheses that can be assessed quantitatively and validated with subsequent observations. 
While we have just begun to use BoSCO and the direct impact on curricular design has yet to be fully 
realized, we have made progress developing a tool that designers will eventually use to inform an 
iterative curricular design process with a more evidenced-based view of factors related to student 
success and retention. 
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